Field of the Invention
This invention relates generally to a system and method for calibrating the cameras in a surround-view camera system on a vehicle and, more particularly, to a system and method for providing online calibration of the cameras in a surround-view camera system on a vehicle.
Discussion of the Related Art
Modern vehicles generally include one or more cameras that provide back-up assistance, take images of the vehicle driver to determine driver drowsiness or attentiveness, provide images of the road as the vehicle is traveling for collision avoidance purposes, provide structure recognition, such as roadway signs, etc. Other vehicle vision applications include vehicle lane sensing systems to sense the vehicle travel lane and drive the vehicle in the lane-center. Many of these known lane sensing systems detect lane-markers on the road for various applications, such as lane departure warning (LDW), lane keeping (LK), lane centering (LC), etc., and have typically employed a single camera, either at the front or rear of the vehicle, to provide the images that are used to detect the lane-markers.
It has been proposed in the art to provide a surround-view camera system on a vehicle that includes a front camera, a rear camera and left and right side cameras, where the camera system generates a top-down view of the vehicle and surrounding areas using the images from the cameras, and where the images overlap each other at the corners of the vehicle. The top-down view can be displayed for the vehicle driver to see what is surrounding the vehicle for back-up, parking, etc. Future vehicles may not employ rearview mirrors, but may instead include digital images provided by the surround view cameras.
U.S. Patent Application Publication No. 2013/0293717 to Zhang et al., filed Apr. 9, 2013, titled, Full Speed Lane Sensing With A Surrounding View System, assigned to the assignee of this application and herein incorporated by reference, discloses a system and method for providing lane sensing on a vehicle by detecting roadway lane-markers, where the system employs a surround-view camera system providing a top-down view image around the vehicle. The method includes detecting left-side and right-side lane boundary lines in the top-down view image, and then determining whether the lane boundary lines in the image are aligned from one image frame to a next image frame and are aligned from image to image in the top-down view image.
For many camera-based vehicle applications it is critical to accurately calibrate the position and orientation of the camera with respect to the vehicle. Camera calibration generally refers to estimating a number of camera parameters including both intrinsic and extrinsic parameters, where the intrinsic parameters include focal length, optical center, radial distortion parameters, etc., and the extrinsic parameters include camera location, camera orientation, etc. Camera extrinsic parameters calibration typically involves determining a set of parameters that relate camera image coordinates to vehicle coordinates and vice versa. Some camera parameters, such as camera focal length, optical center, etc., are stable, while other parameters, such as camera orientation and position, are not. For example, the height of the camera depends on the load of the vehicle, which will change from time to time.
In the known surround-view camera systems, the images from the cameras overlap at the corners of the vehicle, where the camera calibration process “stitches” the adjacent images together so that common elements in the separate images directly overlap with each other to provide the desired top-down view. During manufacture of the vehicle, these camera images are stitched together to provide this image using any of a number of calibration techniques so that when the vehicle is first put into service, the cameras are properly calibrated. One calibration technique employed is to position the vehicle on a checker-board pattern of alternating light and dark squares where each point of the squares is suitably identified. Using these points in the developed images allows the camera calibration software to correct the position of the images so that overlapping points in adjacent images are identified at the same location.
However, once the vehicle is put into service various things may occur that could cause the orientation and position of the cameras to change, where the calibration of the camera includes errors causing misalignment in the top-down image. These things may include loading of the vehicle that causes camera position, such as height, and/or camera orientation, such as pitch, to change relative to world coordinates, small impacts to the vehicle which may change the position and orientation of the cameras, etc. However, current video processing modules (VPM) that process the images from the cameras to generate the top-down view are unable to recalibrate the cameras online while the vehicle is in use. Contrary, the vehicle operator must take the vehicle to a dealer or other authorized service center that has the ability to recalibrate the cameras in the same manner as was done during vehicle manufacture, which has obvious drawbacks.